Method and System for High Performance Data Metatagging and Data Indexing Using Coprocessors

ABSTRACT

Disclosed herein is a method and system for hardware-accelerating the generation of metadata for a data stream using a coprocessor. Using these techniques, data can be richly indexed, classified, and clustered at high speeds. Reconfigurable logic such a field programmable gate arrays (FPGAs) can be used by the coprocessor for this hardware acceleration. Techniques such as exact matching, approximate matching, and regular expression pattern matching can be employed by the coprocessor to generate desired metadata for the data stream.

CROSS-REFERENCE AND PRIORITY CLAIM TO RELATED PATENT APPLICATIONS

This patent application claims priority to U.S. provisional patentapplication 60/865,629 filed Nov. 13, 2006, the entire disclosure ofwhich is incorporated herein by reference.

This patent application is related to U.S. patent application Ser. No.______, filed this same day, entitled “Method and System for HighPerformance Integration, Processing and Searching of Structured andUnstructured Data Using Coprocessors” (identified as Thompson CoburnAttorney Docket Number 44826-72530), the entire disclosure of which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention is generally directed toward the field of metadatageneration and data indexing, particularly the indexing of high speeddata streams.

Terminology:

The following paragraphs provide several definitions for various termsused herein. These paragraphs also provide background informationrelating to these terms.

GPP: As used herein, the term “general-purpose processor” (or GPP)refers to a hardware device having a fixed form and whose functionalityis variable, wherein this variable functionality is defined by fetchinginstructions and executing those instructions (for example, an IntelXeon processor or an AMD Opteron processor), of which a conventionalcentral processing unit (CPU) is a common example.Reconfigurable Logic As used herein, the term “reconfigurable logic”refers to any logic technology whose form and function can besignificantly altered (i.e., reconfigured) in the fieldpost-manufacture. This is to be contrasted with a GPP, whose functioncan change post-manufacture, but whose form is fixed at manufacture.

Software: As used herein, the term “software” refers to data processingfunctionality that is deployed on a GPP or other processing devices,wherein software cannot be used to change or define the form of thedevice on which it is loaded. Firmware: As used herein, the term“firmware” refers to data processing functionality that is deployed onreconfigurable logic or other processing devices, wherein firmware maybe used to change or define the form of the device on which it isloaded.

Coprocessor: As used herein, the term “coprocessor” refers to acomputational engine designed to operate in conjunction with othercomponents in a computational system having a main processor (whereinthe main processor itself may comprise multiple processors such as in amulti-core processor architecture). Typically, a coprocessor isoptimized to perform a specific set of tasks and is used to offloadtasks from a main processor (which is typically a GPP) in order tooptimize system performance. The scope of tasks performed by acoprocessor may be fixed or variable, depending on the architecture ofcoprocessor. Examples of fixed coprocessor architectures includeGraphics Processor Units which perform a broad spectrum of tasks andfloating point numeric coprocessors which perform a relatively narrowset of tasks. Examples of reconfigurable coprocessor architecturesinclude reconfigurable logic devices such as Field Programmable GateArrays (FPGAs) which may be reconfigured to implement a wide variety offixed or programmable computational engines. The functionality of acoprocessor may be defined via software and/or firmware.Hardware Acceleration: As used herein, the term “hardware acceleration”refers to the use of software and/or firmware implemented on acoprocessor for offloading one or more processing tasks from a mainprocessor to decrease processing latency for those tasks relative to themain processor.

Enterprise: As used herein, the term “enterprise” refers to any businessorganization or governmental entity that stores and/or processes data(referred to as “enterprise data”) as part of its ongoing operations.

Database: As used herein, the term “database” refers to a persistentdata store with indexing capabilities to expedite query processing.Various database management system (DBMS) implementations might becategorized as relational (RDBMS), object-oriented (OODBMS),hierarchical, etc.; however, the dominant architecture in today'sindustry is a relational, row-column, structured query language(SQL)-capable database. An ANSI-standard SQL database engine is a maturesoftware architecture that can retrieve structured data in response to aquery, usually in an efficient manner.Structured Data As used herein, the term “structured data” refers todata that has been normalized and persisted to a relational database.Normalization is the data design process of putting data into a tabular,row-column format and abstracting duplicate data into separate tables.Structured data in relational columns is capable of being indexed withB-tree indexes, significantly speeding access to the data in thesecolumns. In SQL terms, structured columns have size limits. Thesecolumns may have constraints and referential integrity applied to themin order to ensure consistent data quality. Examples of commonstructured SQL datatypes are: INT(eger), NUMBER, CHAR(acter), VARCHAR,DATE, TIMESTAMP.Unstructured Data: As used herein, the term “unstructured data” refersto data that falls outside the scope of the definition above forstructured data. Thus, the term unstructured data encompasses files,documents or objects with free form text or embedded values includedtherein. This data includes the complete set of bytes, often includingbinary-format data, that was used by the application that generated it.Examples of unstructured data include word processing documents (e.g.,Microsoft Word documents in their native format), Adobe Acrobatdocuments, emails, image files, video files, audio files, and otherfiles in their native formats relative to the software application thatcreated them. In SQL terms, unstructured columns have very large, if notunlimited size. Common examples of unstructured SQL datatypes are: BLOB,TEXT, XML, RAW, and IMAGE. Unstructured objects may also be storedoutside the database, for example in operating system files. Access tothese external objects from within the database engine uses links in themetadata in the database table to the storage location.

There are a number of reasons why XML is will not normally becategorized as “structured” as that term is used herein:

-   -   XML may have large or unlimited sized values    -   XML often does not have strongly enforced datatyping    -   XML has a flexible schema    -   XML values in elements and attributes is often not as rigidly        conformed and carefully cleansed as traditional “structured”        database columns        Although the concept of “semi-structured” data with flexible        schemas is emerging, particularly for XML, for present purposes        everything that is not has not been normalized and persisted to        a relational database will be considered unstructured data. As        such, a column that is of the XML datatype would thus fall under        this present definition of “unstructured data”.        Metadata: As used herein, the term “metadata” in the context of        data objects and documents refers to data that describes or        characterizes a data object or document. Examples of object and        document metadata include but are not limited to file type, byte        size, date created, date last modified, author, title,        information about the document's/object's data source        (optionally including the name and version number of the program        used to generate the document), information about whether the        data matches other data, subject coverage, classification        information (e.g. information about its concepts,        person/place/entity names that are found within the        document/data object, word counts, etc.), position information        relating to data within a document/object, or other        content-derived information about documents/objects.        Bus: As used herein, the term “bus” refers to a logical bus        which encompasses any physical interconnect for which devices        and locations are accessed by an address. Examples of buses that        could be used in the practice of the present invention include,        but are not limited to the PCI family of buses (e.g., PCI-X and        PCI-Express) and HyperTransport buses.        Pipelining: As used herein, the terms “pipeline”, “pipelined        sequence”, or “chain” refer to an arrangement of application        modules wherein the output of one application module is        connected to the input of the next application module in the        sequence. This pipelining arrangement allows each application        module to independently operate on any data it receives during a        given clock cycle and then pass its output to the next        downstream application module in the sequence during another        clock cycle.        Full-Text Search: As used herein, the term “full-text search”        refers to scanning through the entire body of a document or        object, considering every word or byte. This processing may        allow for approximations, tokenization based on tagging for a        flexible schema, wild-carding, or complex matching.        Text Analytics and Text Mining: As used herein, the terms “text        analytics” and “text mining” refer to algorithms that operate on        document objects using complex language concepts such as        semantics. Examples of text analytic/text mining processing        include: named entity recognition, content extraction, document        classification, document summarization, natural language        processing, statistical pattern learning, and relevance ranking.

BACKGROUND OF THE INVENTION

Enterprises such as corporations, institutions, agencies, and otherentities have massive amounts of data that they need to manage. Whilesome of an enterprise's critical data are normalized, structured, andstored in relational databases, most enterprises' data (generallythought to be around 80% of enterprise data) is unstructured. Withconventional computing systems, effective management and efficientaccess to such unstructured data is problematic.

Indexing is a well-known technique that is used to increase theefficiency by which data can be searched. An index is a list of termsand pointers associated with a collection of data. An example of such anindex 100 is shown in FIG. 1. Index 100 comprises a plurality of indexentries 102, with each index entry 102 comprising a term 104 (see the“term” column in the table) and one or more pointers 106 (see the“pointer(s)” column in the table). The terms 104 in an index can bewords, phrases, or other information associated with the data. In manysituations, these terms are user-specified. Each pointer 106 in an indexcorresponds to the term 104 for that entry 102 and identifies where thatterm can be found in the data. With unstructured data, the datacollection often comprises a plurality of documents. Examples ofdocuments include items such a word processing files, spreadsheet files,emails, images, Adobe Acrobat files, web pages, books, pages of books,etc.

However, the inventors note their belief that conventional indexingtechniques require a tremendous amount of time to generate an effectiveindex. Even relatively small data sets can take days to effectivelyindex with conventional indexing techniques deployed in software oncentral processors such as GPPs because of indexing'scomputationally-intensive nature. Because of the sheer volume of datathat enterprises encounter on a daily basis, it is simply not practicalfor enterprises to index all of the data in its possession (and to whichit has access) using these conventional indexing techniques. Instead,enterprises are forced to make a priori decisions as to which data willbe subjected to indexing; this is particularly true for unstructureddata which comprises the bulk of most enterprises' data. In doing so,enterprises are left without an effective means for efficiently managingand searching much of its data.

SUMMARY OF THE INVENTION

In an effort to address this need in the art, the inventors hereindisclose a technique for hardware-accelerating the generation ofmetadata for data. This data may comprise both structured and/orunstructured data. From this metadata, rich indexes can be generated toenable robust and high performance data searching and analysis. Withembodiments of the present invention, data is streamed into acoprocessor, and metadata for the indexes is generated at bus bandwidthrates, thereby leading to dramatic improvements in indexing latency.Because of such improvements in latency, much larger amounts (if notall) of an enterprise's data can be efficiently and effectively indexed.

In doing so, the present invention preferably harnesses the underlyinghardware-accelerated technology disclosed in the following patents andpatent applications: U.S. Pat. No. 6,711,558 entitled “AssociatedDatabase Scanning and Information Retrieval”, U.S. Pat. No. 7,139,743entitled “Associative Database Scanning and Information Retrieval usingFPGA Devices”, U.S. Patent Application Publication 2006/0294059 entitled“Intelligent Data Storage and Processing Using FPGA Devices”, U.S.Patent Application Publication 2007/0067108 entitled “Method andApparatus for Performing Biosequence Similarity Searching”, U.S. PatentApplication Publication ______ entitled “Method and Apparatus forProtein Sequence Alignment Using FPGA Devices” (published from U.S.application Ser. No. 11/836,947, filed Aug. 10, 2007), U.S. PatentApplication Publication 2007/0130140 entitled “Method and Device forHigh Performance Regular Expression Pattern Matching”, U.S. PatentApplication Publication ______ entitled “Method and Apparatus forApproximate Pattern Matching” (published from U.S. application Ser. No.11/381,214, filed May 2, 2006), U.S. Patent Application Publication2007/0174841 entitled “Firmware Socket Module for FPGA-Based PipelineProcessing”, and U.S. Patent Application Publication 2007/0237327entitled “Method and System for High Throughput Blockwise IndependentEncryption/Decryption”), the entire disclosures of each of which areincorporated herein by reference.

As documents are streamed into a coprocessor, any of a number ofmetadata generation operations can be flexibly performed on the documentstream. For example, exact matching and approximate matching operationscan be performed on the words within a document stream to find whichwords within the document stream are “hits” against the words in any ofa number of dictionaries. From these “hits”, rich indexes can quickly begenerated.

For ease of reference, the term “documents” will be often be used todescribe the unstructured data that is indexed in accordance withvarious embodiments of the present invention. However, it should benoted that the use of the term “documents” is meant to be only exemplaryas other forms unstructured data can also be indexed using thetechniques described herein. Also, the term “words” is used to describebits within a data stream that are grouped together for the purposes ofmetadata generation operations. Words preferably correspond to the bytesthat form discrete character strings, wherein the characters may be anycombination of letters, numbers, or other characters.

In accordance with an extremely powerful aspect of an embodiment of theinvention, the coprocessor can perform regular expression patternmatching on a received document stream to detect whether any of aplurality of pre-defined patterns exist within the document stream.Examples of patterns that can detected using regular expression patternmatching include social security numbers, credit card numbers, telephonenumbers, email address, etc. By developing indexes based on thesepatterns, an enterprise can effectively answer an indefinitepattern-based question such as “How many of our documents contain acredit card number?” without having to re-analyze all of theenterprise's documents. Instead, the generated index pertaining to thecredit card pattern can be accessed to lookup all documents that havebeen identified as containing a credit card number.

Furthermore, classification and clustering operations can be performedon documents using a coprocessor to further analyze and index thosedocuments.

The high performance stream indexing that is enabled by the presentinvention has a myriad of useful applications. For example, the emailtraffic within and/or incoming to an enterprise can be efficientlyindexed “on the fly” before or as that email traffic reaches itsaddressees.

Similarly, the web traffic (e.g., the web pages downloaded by usercomputers within an enterprise) can also be indexed “on the fly” usingvarious embodiments of the present invention.

Web search engines could employ the embodiments of the present inventionto stream web page content through an appliance configured to performthe metadata generation and index generation techniques describedherein. The web search engine can then apply search queries against thegenerated index(es) when formulating responses to those search queries.It is expected that the present invention can be employed todramatically reduce the lag time that it takes for new web pages tobecome “findable” through Internet search engines.

Also, the embodiments of the present invention can be employed tooperate on data being saved to a disk by any computers within anenterprise.

Another application pertains to analyzing and indexing data streams suchas incoming sales transactions, incoming news wire information, and logfiles.

Yet another application pertains to retroactively indexing existingenterprise data, including indexing previously-unindexed enterprise dataand re-indexing previously-indexed enterprise data.

These and other features and advantages of the present invention will beapparent to those having ordinary skill in the art upon review of thefollowing description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary index;

FIG. 2( a) illustrates an exemplary metadata generating appliance inaccordance with an embodiment of the present invention;

FIG. 2( b) illustrates an exemplary metadata generating appliance inaccordance with another embodiment of the present invention;

FIG. 2( c) illustrates an exemplary metadata generating appliance inaccordance with yet another embodiment of the present invention;

FIGS. 3( a) and (b) illustrate exemplary printed circuit boards for usein the appliances of FIGS. 2( a)-(c);

FIG. 4 illustrates an example of how a firmware pipeline can be deployedacross multiple reconfigurable logic devices;

FIG. 5( a) is a high level block diagram view of how a coprocessor canbe used to generate metadata for data that streams therethrough;

FIG. 5( b) depicts an exemplary index that can be created from themetadata produced by the coprocessor of FIG. 5( a);

FIG. 6 depicts an exemplary firmware pipeline that can be deployedwithin reconfigurable logic to perform a desired metadata generationoperation;

FIG. 7( a) depicts an exemplary firmware pipeline that can be used toperform the stream pre-processing operation shown in FIG. 6;

FIG. 7( b) depicts an exemplary output stream of words and featurevectors that can be produced by the firmware pipeline of FIG. 7( a);

FIG. 8( a) depicts an exemplary specialized index generation operationthat performs seeks to exactly match the words within the incoming datastream against the words in a dictionary to thereby generate aspecialized index;

FIG. 8( b) depicts an exemplary specialized index generation operationthat performs seeks to approximately match the words within the incomingdata stream against the words in a dictionary to thereby generate aspecialized index;

FIG. 9( a) depicts an exemplary specialized index generation operationthat performs seeks to exactly match the words within the incoming datastream against the words in a plurality of different dictionaries tothereby generate a plurality of specialized indexes;

FIG. 9( b) depicts an exemplary specialized index generation operationthat performs seeks to match the words within the incoming data streamagainst a plurality of different regular expression patterns to therebygenerate a plurality of specialized indexes;

FIG. 9( c) depicts an exemplary specialized index generation operationthat performs seeks to match the words within the incoming data streamagainst the words in a plurality of different dictionaries and against aplurality of different regular expression patterns to thereby generate aplurality of specialized indexes;

FIG. 10 depicts an exemplary technique for inserting the generatedindexes into operational indexes maintained by a relational database;

FIG. 11( a) depicts a stream pre-processing firmware pipeline thatincludes a word counting module;

FIG. 11( b) depicts an exemplary index that includes metadata relatingto the word counting function of FIG. 11( a);

FIG. 11( c) depicts an exemplary firmware pipeline wherein the generatedmetadata corresponds to classification data about the incomingdocuments;

FIG. 12( a) depicts an exemplary classification operation that producesclass scores for incoming documents to assess their relevance to aplurality of different classifications;

FIG. 12( b) depicts an exemplary classification operation thatdetermines a classification for each incoming document;

FIGS. 13( a) and (b) depict exemplary firmware pipelines that employhistogram generation to enable document clustering;

FIGS. 14( a) and (b) depict exemplary firmware pipelines that performclassification, specialized index generation, and histogram generationoperations on incoming documents;

FIG. 15 depicts an exemplary environment in which the appliance of FIGS.2( a)-(c) can be employed;

FIG. 16 illustrates an exemplary overview of a document ingestpre-processing operation that can employ the metadata generationtechniques described herein;

FIG. 17 illustrates an exemplary logical view of a document ingestpre-processing operation in accordance with the FIG. 2( c) embodiment;and

FIG. 18 illustrates an exemplary data flow within the appliance of FIG.2( c) for a document ingest pre-processing operation that can employ themetadata generation techniques described herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 2( a) depicts an exemplary embodiment for a metadata generatingappliance 200. While an embodiment of appliance 200 can be referred toas a metadata generating appliance, it should be noted thatfunctionalities in addition to metadata generation can be supported byappliance 200, as explained in the above-referenced and incorporatedU.S. patent application Ser. No. ______, filed this same day, entitled“Method and System for High Performance Integration, Processing andSearching of Structured and Unstructured Data Using Coprocessors”(identified as Thompson Coburn Attorney Docket Number 44826-72530).

Preferably, appliance 200 employs a hardware-accelerated data processingcapability through coprocessor 450 to generate the desired metadata.Within appliance 200, a coprocessor 450 is positioned to receive datathat streams into the appliance 200 from a network 420 (via networkinterface 410). Network 420 preferably comprises an enterprise network(whether LAN or WAN), in which various disparate data sources arelocated. Such incoming data may comprise both structured andunstructured data as appliance 200 can provide beneficial metadatageneration operations for both data types.

The computer system defined by processor 412 and RAM 408 can be anycommodity computer system as would be understood by those havingordinary skill in the art. For example, the computer system may be anIntel Xeon system or an AMD Opteron system. Thus, processor 412, whichserves as the central or main processor for appliance 200, preferablycomprises a GPP.

In a preferred embodiment, the coprocessor 450 comprises areconfigurable logic device 402. Preferably, data streams into thereconfigurable logic device 402 by way of system bus 406, although otherdesign architectures are possible (see FIG. 3( b)). Preferably, thereconfigurable logic device 402 is a field programmable gate array(FPGA), although this need not be the case. System bus 406 can alsointerconnect the reconfigurable logic device 402 with the appliance'sprocessor 412 as well as the appliance's RAM 408. In a preferredembodiment, system bus 406 may be a PCI-X bus or a PCI-Express bus,although this need not be the case.

The reconfigurable logic device 402 has firmware modules deployedthereon that define its functionality. The firmware socket module 404handles the data movement requirements (both command data and targetdata) into and out of the reconfigurable logic device, thereby providinga consistent application interface to the firmware application module(FAM) chain 350 that is also deployed on the reconfigurable logicdevice. The FAMs 350 i of the FAM chain 350 are configured to performspecified data processing operations on any data that streams throughthe chain 350 from the firmware socket module 404. Preferred examples ofFAMs that can be deployed on reconfigurable logic in accordance with apreferred embodiment of the present invention are described below.

The specific data processing operation that is performed by a FAM iscontrolled/parameterized by the command data that FAM receives from thefirmware socket module 404. This command data can be FAM-specific, andupon receipt of the command, the FAM will arrange itself to carry outthe data processing operation controlled by the received command. Forexample, within a FAM that is configured to perform an exact matchoperation, the FAM's exact match operation can be parameterized todefine the key(s) that the exact match operation will be run against. Inthis way, a FAM that is configured to perform an exact match operationcan be readily re-arranged to perform a different exact match operationby simply loading new parameters for one or more different keys in thatFAM.

Once a FAM has been arranged to perform the data processing operationspecified by a received command, that FAM is ready to carry out itsspecified data processing operation on the data stream that it receivesfrom the firmware socket module. Thus, a FAM can be arranged through anappropriate command to process a specified stream of data in a specifiedmanner. Once the FAM has completed its data processing operation,another command can be sent to that FAM that will cause the FAM tore-arrange itself to alter the nature of the data processing operationperformed thereby. Not only will the FAM operate at hardware speeds(thereby providing a high throughput of data through the FAM), but theFAMs can also be flexibly reprogrammed to change the parameters of theirdata processing operations.

The FAM chain 350 preferably comprises a plurality of firmwareapplication modules (FAMs) 350 a, 350 b, . . . that are arranged in apipelined sequence. However, it should be noted that within the firmwarepipeline, one or more parallel paths of FAMs 350 i can be employed. Forexample, the firmware chain may comprise three FAMs arranged in a firstpipelined path (e.g., FAMs 350 a, 350 b, 350 c) and four FAMs arrangedin a second pipelined path (e.g., FAMs 350 d, 350 e, 350 f, and 350 g),wherein the first and second pipelined paths are parallel with eachother. Furthermore, the firmware pipeline can have one or more pathsbranch off from an existing pipeline path. A practitioner of the presentinvention can design an appropriate arrangement of FAMs for FAM chain350 based on the processing needs of a given application.

A communication path 430 connects the firmware socket module 404 withthe input of the first one of the pipelined FAMs 350 a. The input of thefirst FAM 350 a serves as the entry point into the FAM chain 350. Acommunication path 432 connects the output of the final one of thepipelined FAMs 350 m with the firmware socket module 404. The output ofthe final FAM 350 m serves as the exit point from the FAM chain 350.Both communication path 430 and communication path 432 are preferablymulti-bit paths.

The nature of the software and hardware/software interfaces used byappliance 200, particularly in connection with data flow into and out ofthe firmware socket module are described in greater detail in theabove-referenced and incorporated U.S. Patent Application Publication2007/0174841.

FIG. 2( b) depicts another exemplary embodiment for appliance 200. Inthe example of FIG. 2( b), appliance 200 includes a relational databasemanagement system 304 that is in communication with bus 406 via diskcontroller 414. Thus, the data that is streamed through the coprocessor450 may also emanate from RDBMS 304. Furthermore, the metadata generatedby coprocessor 450 can be stored as structured data within RDBMS 304from which it can be used as an index for various datasearching/analysis operations. Such indexes can take the form of B-treeindexes. An example of such usage for the appliance 200 of FIG. 2( b) isdescribed in the above-referenced and incorporated U.S. patentapplication Ser. No. ______, filed this same day, entitled “Method andSystem for High Performance Integration, Processing and Searching ofStructured and Unstructured Data Using Coprocessors” (identified asThompson Coburn Attorney Docket Number 44826-72530).

FIG. 2( c) depicts another exemplary embodiment for appliance 200. Inthe example of FIG. 2( c), appliance 200 also includes a data store 306of unstructured data that is in communication with bus 406 via diskcontroller 416. Thus, the data that is streamed through the coprocessor450 may also emanate from data store 306. Furthermore, the unstructureddata that is streamed through coprocessor 450 to generate metadatatherefor can optionally be stored within data store 306. As described inthe above-referenced and incorporated U.S. patent application Ser. No.______, filed this same day, entitled “Method and System for HighPerformance Integration, Processing and Searching of Structured andUnstructured Data Using Coprocessors” (identified as Thompson CoburnAttorney Docket Number 44826-72530), the metadata tables within RDBMS304 can be used to render data search/analysis operations on theunstructured data stored within data store 306 more efficient.

FIG. 3( a) depicts a printed circuit board or card 700 that can beconnected to the PCI-X or PCI-e bus 406 of a commodity computer systemfor use as a coprocessor 450 in appliance 200 for any of the embodimentsof FIGS. 2( a)-(c). In the example of FIG. 3( a), the printed circuitboard includes an FPGA 402 (such as a Xilinx Virtex II FPGA) that is incommunication with a memory device 702 and a PCI-X bus connector 704. Apreferred memory device 702 comprises SRAM and DRAM memory. A preferredPCI-X or PCI-e bus connector 704 is a standard card edge connector.

FIG. 3( b) depicts an alternate configuration for a printed circuitboard/card 700. In the example of FIG. 3( b), a bus 706 (such as a PCI-Xor PCI-e bus), one or more disk controllers 708, and a disk connector710 are also installed on the printed circuit board 700. Any commoditydisk interface technology can be supported, as is understood in the art.In this configuration, the firmware socket 404 also serves as a PCI-X toPCI-X bridge to provide the processor 412 with normal access to anydisk(s) connected via the private PCI-X bus 706. It should be noted thata network interface can be used in addition to or in place of the diskcontroller and disk connector shown in FIG. 3( b).

It is worth noting that in either the configuration of FIG. 3( a) or3(b), the firmware socket 404 can make memory 702 accessible to the bus406, which thereby makes memory 702 available for use by an OS kernel asthe buffers for transfers to the FAMs from a data source with access tobus. It is also worth noting that while a single FPGA 402 is shown onthe printed circuit boards of FIGS. 3( a) and (b), it should beunderstood that multiple FPGAs can be supported by either including morethan one FPGA on the printed circuit board 700 or by installing morethan one printed circuit board 700 in the appliance 200. FIG. 4 depictsan example where numerous FAMs in a single pipeline are deployed acrossmultiple FPGAs.

FIG. 5( a) depicts at a high level a coprocessor 450 that receives anincoming data stream, generates metadata for the data stream, andoutputs the data stream together with the generated metadata. Examplesof different hardware-accelerated metadata generation operations will bedescribed in greater detail hereinafter. Any of a number of text miningand text analytic operations can be employed to generate metadata. Someexamples of metadata generation operations include but are not limitedto part-of-speech tagging, information and entity extraction, documentclassification, document clustering, and text summarization. In theseexamples, the data stream under consideration will be a stream ofdocuments. As noted, this usage in only exemplary as data streams otherthan documents can be readily processed using coprocessor 450 and thetechniques described herein. These data streams may compriseunstructured and/or structured data.

FIG. 5( b) depicts an exemplary index 100 that can built from themetadata generated by coprocessor 450. As described in connection withFIG. 1, each table entry 102 comprises a term 104 and its associatedpointer 106. In the example of FIG. 5( b), each pointer 106 comprises adocument identifier D_(i) and one or more position identifiers p_(i).Each document identifier identifies a document in which the term 104corresponding to that pointer 106 is present. Each position identifierin the pointer identifies a position within the identified documentwhere that term 104 is located. Preferably, the position identifier isrepresented by a byte offset relative to the start of a document. Thus,the pointers for the term “Azkaban” in FIG. 5( b) establish that“Azkaban” can be found in Document D₁₂ at position p₁ and in DocumentD₉₉₁₉ at positions p₁ and p₂. Any of a number of well-known techniquescan be used to map pointers such as the ones shown in FIG. 5( b) to thedocuments stored within a file system of an enterprise.

FIG. 6 depicts exemplary firmware 350 that can be employed inreconfigurable logic 402 to perform a metadata generation operation 604on an incoming stream of documents 600. An example of a metadatageneration operation 604 includes a stream pre-processing operation 602.The stream pre-processing preferably comprises various position andfeature vector extraction operations. These operations may employtechniques known in the art as cryptographic hashing to provide the termdescriptor and tables to enables the values and lists described hereinto be generated and exploited.

FIG. 7( a) depicts an exemplary sequence of processing modules for thestream pre-processing stage 602. A word parsing module 750 preferablyoperates to identify white-space and/or punctuation delimited terms inthe documents and record their positions. These recorded positions canthen be used to help determine word positions within documents (e.g.,the p_(i) values in the pointers of FIG. 5( b)). A stop list filtermodule 752 preferably operates to remove words from the document streamthat appear on a stop list. This may be desirable to remove common words(e.g. “the”, “a”, “an”) for which indexing may not be particularlyhelpful from the document stream. The stop list is preferably adjustableto control which words will be removed from the document stream. Astemming module 754 preferably operates to stem words in the documentstream to their roots. Examples of stemming operations include removingplurals, removing prefixes, removing suffixes, removing gerunds, etc. Itshould also be noted that other processing modules can be included instream pre-processing module 602 if desired. For example, aspell-checking module can be used to correct possible spelling errorsthat exist within a word stream.

FIG. 7( b) depicts an exemplary output data stream from the streampre-processing operation 602, wherein the stream includes the parsedwords 760 such that each parsed word is accompanied by a positionidentifier 758 for that word. Furthermore, a document identifier 756 ispresent that identifies when a new document is present within thestream. While the example of FIG. 7( b) shows a serial stream of suchdata, it should be noted that the document identifiers 756 and positionidentifiers 758 can flow in bit streams within firmware 350 that areparallel to a bit stream of words 760.

Returning to FIG. 7( a), as words stream through the pre-processingmodule 602, a memory containing an index for the documents can begenerated. This index 100 can serve as a general index that is populatedwith entries for each different word encountered by the streampre-processing module 602 together with its associated position vectorinformation (e.g., document identifier and position identifier as shownin FIG. 5( b)). If a word that has already been added to general index100 is encountered again within the stream, then the pointer informationin index 100 for that word can be updated to reflect the latest positionat which the word has been found.

It should be noted that the memory used to store the index 100 as wordsstream through the pre-processing module 602 can be located in any of anumber of places. For example, this memory can be located in theavailable on-chip memory of the FPGA 402. This memory could also bememory device 702 shown in FIGS. 3( a) and (b). Further still, thismemory could be RAM 408. This memory could even be a location such asRDBMS 304. Preferably, coprocessor 450 performs the action of updatingindex 100. Techniques such as hashing can be used to insert new datainto the index 100.

It should be noted that each of the modules shown in FIG. 7( a) as wellas the stream pre-processing operation 602 itself is an optionalfeature. If desired, the metadata generation operation 604 can beperformed on documents 600 that have not been pre-processed.

FIG. 8( a) depicts an exemplary embodiment for the metadata generationoperation 604 wherein a specialized index generation operation 850 isalso performed. In FIG. 8( a), words and position vectors stream into anexact matching module 800. This exact matching module 800 is loaded withthe words 804 from a dictionary 802. The exact matching modules will usewords 804 as keys against which the streaming words are compared todetermine whether any exact matches exist therebetween. As matches arefound by module 800, a specialized index 860 maintained in the memory isupdated with the matching word information and the correspondingpointers for those matching words (see FIG. 5( b)). The position vectorsfound by the stream pre-processing operation 602 (e.g., documentidentifiers 756 and position identifiers 758) can be used as the pointerinformation. Thus, in addition to the general index 100 described inconnection with FIG. 7( a), a specialized index 860 (that is specific asto dictionary 802) can also be efficiently generated via the dataprocessing operations of coprocessor 450.

Any of a number of exact matching techniques can be used to perform theexact matching operation. For example, the hardware-accelerated matchingtechniques disclosed in the above-referenced and incorporated U.S. Pat.Nos. 6,711,558 and 7,139,743 and U.S. Patent Application Publications2006/0294059, 2007/0130140, and U.S. Patent Application Publication______ entitled “Method and Apparatus for Approximate Pattern Matching”(published from U.S. application Ser. No. 11/381,214, filed May 2,2006).

Any collection of words can be used as dictionary 802. Examples ofdictionaries 802 that can be used include large word collections such asfull English language dictionaries, full foreign language dictionaries,scientific/technical dictionaries, etc. Smaller word collections canalso be used.

FIG. 8( b) depicts an alternate embodiment for index generation whereinan approximate matching module 810 is used rather than an exact matchingmodule. It may be desirable to employ an approximate matching module 810for the index generation operation 850 to reduce the likelihood thatspelling errors and the like do not result in document text beingmis-indexed. Examples of hardware-accelerated approximate matchingtechniques suitable for use as approximate matching module 810 are alsodisclosed in the above-referenced and incorporated U.S. Pat. Nos.6,711,558 and 7,139,743 and U.S. Patent Application Publications2006/0294059, 2007/0130140, and U.S. Patent Application Publication______ entitled “Method and Apparatus for Approximate Pattern Matching”(published from U.S. application Ser. No. 11/381,214, filed May 2,2006). It should be noted that if a spell-checking module as describedabove is used in the stream pre-processing stage, it may be unnecessaryto employ an approximate matching module to accommodate mis-spellingsand the like.

It should also be noted that a plurality of exact matching/approximatematching modules 800/810 can be deployed in parallel to distribute theworkload of matching streaming words to the word keys 804 acrossmultiple matching modules 800/810. However, if desired, a single serialpipelined sequence of compare registers can be used by an matchingmodule 800/810 to sequentially compare the streaming words with wordkeys 804 if desired.

It should be noted that, like the memory used to store general index100, the memory used to store the specialized indexes 860 can be locatedin any of a number of places (e.g., the on-chip memory of the FPGA 402,memory device 702, etc.).

A powerful aspect of various embodiments of the present invention is theability to concurrently generate multiple specialized indexes. Anexample of this is shown in FIG. 9( a). With this embodiment, thematching module 800/810 matches the incoming words against multipledictionaries 802 _(i), each dictionary 802 _(i) having its owncollection of words which will serve as the keys against which theincoming words are judged. As shown, the words within dictionaries suchas an English dictionary, a French dictionary, a medical dictionary, anda technical dictionary can be compared against the incoming word stream.As matches are found between the incoming words and the words within anyof these dictionaries, a specialized index associated with each of thesedictionaries can be updated with pointer information. FIG. 9( a) depictsan English dictionary index 902 and a technical dictionary index 904 asexamples of such specialized indexes. Preferably, these specializedindexes are pre-populated in the term column with all of the wordswithin its associated dictionary. As that term is found within the wordstream by the matching module 800/810, the pointer information for thatterm can be updated as appropriate.

The general index 100 in such an embodiment can be updated when wordsstreaming through the exact matching module do not find a match in anyof the dictionaries 802. Furthermore, the entries in the otherspecialized indexes can be merged into the general index if desired tothereby make general index 100 applicable to all of the words withindocument stream 600. This merger can be performed offline orconcurrently with the updating of the other indexes. Otherwise, thecoprocessor 450 can be configured to update the general index 100 aswords stream out of the stream pre-processing module 602 and update thespecialized indexes as words stream out of the exact/approximatematching module 800/810.

Another powerful aspect of various embodiments of the present inventionis the ability to perform regular expression pattern matching onincoming words. An example of this is shown in FIG. 9( b). Regularexpressions, as is well-known in the art, can be used to detect patternsof varying degrees of complexity within a word stream. An exemplarypattern for which regular expressions are useful means of detection is asocial security number (SSN). SSNs exhibit the pattern xxx-xx-xxxx,wherein x can be any number between 0 and 9. Items such as phonenumbers, credit card numbers, primary account numbers (PANs), e-mails,URLs, and others also exhibit patterns that are detectable throughregular expressions.

As shown in FIG. 9( b), a regular expression matching module 910 can bedeployed on coprocessor 450 (preferably in firmware 350 on areconfigurable logic device 402). The regular expression matching module910 can be configured to detect one or more user-defined patterns. Forexample, the regular expression matching module 910 in FIG. 9( b) isconfigured to detect words (including word portions) that exhibit ane-mail pattern 912, a phone number pattern 914, a credit card numberpattern 916, and an SSN pattern 918. As these patterns are found withinthe word stream by the regular expression matching module 910, aspecialized index associated with each of these patterns can be updatedwith the content of the matching pattern and with the pointerinformation for that matching pattern. FIG. 9( b) depicts an e-mailindex 920 and a SSN index 922 as examples of such indexes. Thus, if123-45-6789 is encountered in the word stream, then the regularexpression matching module 910 will identify this pattern as a SSNpattern, and the regular expression matching module 910 will add anentry to the SSN index 922 comprising the term “123-45-6789” and apointer to that term in the word stream. Preferably, as described inconnection with FIG. 9( a), a general index 100 is also updated as wordsstream through the regular expression matching module 910, and theregular expression matching module 910 does not find any patternmatches.

An example of a hardware-accelerated regular expression matchingtechnique suitable for use as regular expression matching module 910 isdisclosed in the above-referenced and incorporated U.S. PatentApplication Publication 2007/0130140.

Also, as noted above in connection with the matching modules 800/810,various combinations of sequential and parallel regular expressionmatching modules 910 can be employed within coprocessor 450 todistribute the pattern matching workload across multiple resources.

The power of such pattern indexes shown by FIG. 9( b) cannot beunderstated. It is typically extremely difficult for a person within anenterprise to efficiently get an answer to a question such as “Whichdocuments within our enterprise include a SSN?” or a command such as“Show me all documents within our enterprise that include a credit cardnumber”. Similarly, it is difficult for a web search engine to providerelevant results to an Internet search query seeking to find web pageswhich contain credit card numbers or SSNs without being specific as toindividual numbers. However, through the indexing techniques disclosedin connection with FIG. 9( b), the answers to questions such as theseare readily available.

FIG. 9( c) illustrates an exemplary embodiment wherein thehardware-accelerated index generation operation 850 includes one or moreexact/approximate matching modules 800/810 and one or more regularexpression matching modules 910 to generate corresponding specializedindexes (e.g., indexes 902, 904, 920, and 922) and a general index 100.Once again the power of such efficiently-generated indexes cannot beunderstated. These indexes can serve to answer complicated querycommands such as “Show me all documents which include a credit cardnumber and contain the word ‘stolen’ ” without re-analyzing the entireset of documents 600.

It should be noted that coprocessor 450 is preferably configured tochange the dictionaries and/or patterns used by the matching modules800/810/910. Preferably, such changes are made in response to usercommands received as command data by the firmware socket module 404.These changes may comprise adding/removing dictionaries/patterns and/oradding/removing words to/from the various dictionaries.

It should also be noted that the coprocessor 450 can deploy thesemodules 800/810 and 910 in a sequential, parallel, or hybridsequential-parallel arrangement as desired.

FIG. 10 depicts an embodiment wherein periodic batch transfers of thegenerated indexes (e.g., indexes 100, 902, 904, 920, and 922) areperformed to merge them into their corresponding operationalcounterparts within an RDBMS (e.g., RDBMS 304 or some other RDBMS withinenterprise network 420). Thus, batch insert/merge operations can beperformed to insert the content of general index 100 into operationalgeneral index 1000. Similarly, the content of indexes 902, 904, 920, and922 can be inserted into operational indexes 1002, 1004, 1020, and 1022respectively. These transfers can be performed periodically at scheduledintervals or can be performed on an as needed basis as the availablememory space for indexes 100, 902, 904, 920, and 922 runs low.

After the operational indexes have been updated, the newly indexed datacan then be accessed via standardized queries such as SQL commands thatare applied to the operational indexes stored by SQL databases. Theabove-referenced and incorporated U.S. patent application Ser. No.______, filed this same day, entitled “Method and System for HighPerformance Integration, Processing and Searching of Structured andUnstructured Data Using Coprocessors” (identified as Thompson CoburnAttorney Docket Number 44826-72530) discloses a particularly efficientquery handling technique for queries directed toward unstructured datafor which an RDBMS maintains a metadata index.

Another metadata generation operation that can be performed by thestream pre-processing module 602 is a word counting operation 1150, asshown in FIG. 11( a). In order to ascertain the relevance of thedocuments 600 during a query handling operation, the indexing enginepreferably obtains statistical information about the words in thedocuments. This statistical information may includes a count of thenumber of times a particular word appears in a document and a count ofthe number of times a particular word appears in the entire documentcollection (should the documents 600 be grouped into one or morecollections). Another parameter that can be determined and used forrelevance assessments is the size (in words or bytes) of each document.By knowing these parameters, the relevance of a document retrievedduring querying can be calculated using well-known probabilisticrelevance models such as the Okapi BM25 model. See Karen Spärck Jones,Steve Walker, and Stephen E. Robertson. “A Probabilistic Model ofInformation Retrieval: Development and Comparative Experiments (parts 1and 2).”, Information Processing and Management, 36(6):779-840. 2000,the entire disclosure of which is incorporated herein by reference.

Through a word counting module 1150, word lists for documents anddocument collections are generated as the words stream therethrough.Associated with each word on the list will be a count value that isincremented each time the associated word is encountered in the stream.FIG. 11( b) depicts an exemplary index 1154 wherein the pointers 106include a word count parameter wc that identifies how many times eachterm appears in the referenced document. These pointers may also includeparameters that identify the size of each document (although thisinformation can be maintained in a separate index table). Table 1154 mayalso include an additional column 1160 that includes a “collectioncount” parameter that identifies how many times each term appears in adocument collection. Appropriate flags can be added to the bits in theword and position vector stream to identify collection separations.Because the output from the word counting module preferably includesthis count information, FIG. 11( a) depicts the output from the wordcounting module 1150 as including count vectors.

Another metadata generation operation 604 that can be performed bycoprocessor 450 is a classification operation 1100. An exemplarypipeline for such actions is shown in FIG. 11( c). An example of adocument classification operation that can be performed on documentsincludes language classification. With language classification, thedocument can be applied to a statistical n-gram algorithm that isconfigured to identify the language that the text within the documentmost closely matches. See William B. Cavnar and John M. Trenkle,“N-Gram-Based Text Categorization”, Proceedings of SDAIR-94, 3rd AnnualSymposium on Document Analysis and Information Retrieval, Las Vegas,pages 161-175, 1994, the entire disclosure of which is incorporatedherein by reference. Other document classification operations may employhidden Markov models (HMMs) to learn some type of classification for thedocument. See Ludovic Denoyer, Hugo Zaragoza and Patrick Gallinari,“HMM-based Passage Models for Document Classification and Ranking”,Proceedings of ECIR-01, 23rd European Colloquium Information RetrievalResearch, Darmstatd, DE, pages 126-135, 2001, the entire disclosure ofwhich is incorporated herein by reference.

FIG. 12( a) depicts an exemplary classification operation wherein a listof words is maintained for a plurality of different classes. Forexample, a first class 1202 is associated with words 1204 and so on suchthat class n 1206 is associated with words 1208. These words can be usedby one or more matching modules 800/810 as keys that are comparedagainst incoming words. As the key words find a match to a word within adocument, a scoring memory 1210 that tracks a count of matches for eachclass can be updated as appropriate. Thus, when the matching module800/810 finds a match between a given word within Document X and a word1204 within the first class 1202, then the “count” field for class 1 inthe scoring memory 1210 associated with that document can be updated.These counts can serve as class scores for each document to therebyindicate how relevant a given document is to a particular class.Preferably, the scoring memory 1210 is refreshed upon encountering a newdocument in the word stream. Scoring memory 1210 is preferablymaintained in the available on-chip memory of the FPGA 402 or in memorydevice 702 (see FIGS. 3( a) and (b)).

The different classes can pertain to classifications such as “news”,“sports”, “legal”, “medicine”, etc. The words within each class can bewords that are generally expected to be found in a document thatpertains to the subject defined by the class. For example, the word“baseball” may be present in the word list for a “sports”classification, and the word “estop” may be present in the word list fora “legal” classification. Another exemplary document classification canbe language classification. In such an application, each class shown inFIG. 12( a) can be associated with a different language, wherein thewords within each class would comprise a full (or redacted) dictionaryfor that language. With reference to the embodiment of FIG. 9( a), itcan be understood that such language classification operations canfunction as an adjunct to specialized indexing, wherein the matchingoperations shown in FIG. 9( a) can also operate to update a scoringmemory 1210 as appropriate. Yet another exemplary documentclassification can be “reading comprehension level”. Each class can beassociated with a different comprehension level (e.g., “5^(th) gradereading level”, “high school freshman reading level”, “undergraduatereading level”, etc.), and the words within each class can be wordsexpected to be encountered in documents belonging to the correspondingcomprehension level. It should be noted that the count vectorinformation produced by the word counting module of FIG. 11( a) may behelpful in aiding classifications such as “comprehension level”.

FIG. 12( b) depicts an exemplary classification embodiment whereindecision logic 1212 is employed to assign one or more classes to adocument based on the class counts for that document as recorded byscoring memory 1210. For example, each class can have an associatedthreshold value. If the class count meets or exceeds the threshold valuefor its class, then the document can be tagged as belonging to thatclass. As such, it may be possible for the same document to belong tomultiple classes should its counts in those classes meet or exceed theclasses' assigned threshold values.

Furthermore, it should be noted that one or more regular expressionmatching modules 910 can be advantageously employed in connection withclassification operation 1100 (possibly in combination with one or morematching modules 800/810). For example, if a document is found tocontain a credit card number, then an inference could possibly be drawnthat the document should be classified as a “sales record”, “personalrecord”, or the like. Regular expression matching can also be useful togenerate counts of how many words have particular character lengths. Forexample, the regular expression “xxxx” where x can be any character canbe used to update a count of 4 letter words, while the regularexpression “xxxxx” can be used to update a count of 5 letter words, andso on. Such word length counts can be useful for assessingclassifications such as “comprehension level” discussed above.

The classification module 1100 may be used to enrich the word streamemanating from the module 1100 by adding bit strings for each documentto the stream, wherein each bit string tags each document as belongingto a particular class. For example, if the classification module isconfigured to determine which of 20 classifications a document belongsto, a 20 bit string can be added to the output stream for each document,wherein each bit is associated with a particular classification andflags whether the document belongs to that classification. As such, bitX in this bit string can be set aside to flag whether the documentcontains a credit card number. Thus, if a regular expression matchingmodule 910 detects the presence of a credit card number pattern within adocument, the bit string for that document can be updated to set bit Xto high. Similarly, bit Y in the bit string can be set aside to flagwhether the document is a legal document. If the classificationoperation results in a determination that the document should beconsidered a legal document, then bit Y can be set to high.

It should also be noted that classification operation 1100 can beconsidered as also belonging to the specialized index generationcategory of metadata generation operations because the classificationinformation produced by classification operation 1100 can also be usedto generate specialized indexes of documents by class. Such indexes canserve as powerful searching aids in that they can be used to answerquestions such as “How many Spanish language documents are within theenterprise?” and conduct research such as “What relationships existbetween sports and medicine?” and “Find all legal documents in whichJohn Smith is mentioned”.

Also, it should be noted that the classification operation can beperformed in software (e.g., software executed by processor 412) usingthe general and specialized indexes generated from embodiments such asthose shown in FIGS. 9( a)-(c). These different indexes can becross-correlated with pre-defined word lists for the different classesto determine the classification information for each document.

With reference to FIG. 13( a), and following from FIG. 11( a), anothermetadata generation operation 604 that can be performed by coprocessor450 is a histogram generation operation 1300 to thereby aid documentclustering. With document clustering, it is desired to know therelevance of different documents to each other. To aid such clusteringdeterminations, the coprocessor 450 can be configured to perform thehistogram generation operation 1300 on documents 600 (wherein thisoperation can be similar if not identical to the word counting operationdescribed above). These histograms can identify the frequency of wordswithin a document. From the generated histograms, software can be usedto perform feature vector expansion and identify document clusters(operation 1302) that are likely related to each other with respect tocommon subject matter, etc. Feature vector expansion is a standardtechnique wherein the feature vectors of individual documents areexpanded to include words from all the documents in the collection.Clustering can be performed using any of the well-known techniques suchas K-means clustering, fuzzy C-means clustering, hierarchicalclustering, etc. See M. Steinbach, G. Karypis, and V. Kumar. “Acomparison of document clustering techniques.”, KDD Workshop on TextMining, 2000, the entire disclosure of which is incorporated herein byreference.

Should the stream pre-processing module 602 already employ wordcounting, then the histogram generation stage 1300 can be omitted, asshown in FIG. 13( b).

It should be noted that the clustering operations shown by FIGS. 13( a)and (b) can be considered as also belonging to the specialized indexgeneration category of metadata generation operations because theclustering information can also be used to generate specialized indexesof documents by clusters. Such indexes can serve as powerful searchingaids in that they can be used to answer questions such as “Whichdocuments within an enterprise are similar to Document X?” and “Whichbooks available for sale from this website are similar to Book X?”.

FIGS. 14( a) and (b) depict exemplary pipelines wherein thehardware-accelerated metadata generation operation 604 includesclassification 1100, specialized index generation 850, and histogramgeneration 1300 (or word counting 1150).

FIG. 15 depicts an exemplary environment in which appliance 200 can beeffectively employed. FIG. 15 depicts an enterprise computing system1504 that receives a large volume of incoming data from the Internet1500. After this incoming data passes enterprise firewall 1502, it canbe streamed through appliance 200 which taps into the firewall output.Appliance 200 can thus be used to generate metadata and indexes for theincoming data as that data reaches the enterprise and before it lands indata storage somewhere within enterprise network 420. The data processedby appliance 200 can also include data originating from within theenterprise computing system 1504. Furthermore, appliance 200 canoptionally be configured to output its generated metadata for deliveryto other repositories within enterprise network 420. Further still,queries can optionally be applied to appliance 200 to conduct full-textsearching or other data analysis operations on data indexed by appliance200. As noted, an example of such a use for appliance 200 is disclosedin the above-referenced and incorporated U.S. patent application Ser.No. ______, filed this same day, entitled “Method and System for HighPerformance Integration, Processing and Searching of Structured andUnstructured Data Using Coprocessors” (identified as Thompson CoburnAttorney Docket Number 44826-72530).

FIG. 16 depicts an overview of document ingest pre-processing inaccordance with an embodiment where appliance 200 is used to indexdocuments within an enterprise. Preferably, through some form ofdocument ingest GUI 1600 displayed on a user computer within enterprisenetwork 420, the user is able to specify which document(s) should beingested into data store 306 of FIG. 2( c). Optionally, the user canalso key in various forms of metadata about the document(s) to beingested; however this need not be the case as the coprocessor 450 willbe arranged to perform the desired metadata generation operationsautomatically, as described above. In response to an appropriate command1612 delivered to appliance 200 from GUI 1600, one or more documents 600stored in a data store 308 accessible through the enterprise network420, but external to the appliance 200, is delivered to the appliance200. Various adapters can be employed in document retrievalfunctionality 1652 employed by the appliance 200 to provide access todocuments stored on common file systems such as NTFS, FAT, CIFS, variousflavors of Unix file systems, as well as Web access over HTTP.

Should the coprocessor employ a reconfigurable logic device 402 withfirmware 350 deployed thereon, FAMs resident in firmware pipeline 350will be arranged to perform a document metadata generation operation onthe documents it receives. Examples of these metadata generationoperations are described above.

The document metadata 1650 that is produced by the operation of firmware350 can then be stored in RDBMS 304, wherein the RDBMS engine operatesto generate and maintain an index of this document metadata that canlater be queried using standardized database queries to identify whichdocuments within data store 306 should be processed through thecoprocessor 450 at hardware speeds during query processing. After thereceived document 600 has been processed by firmware 350, document 600can then be ingested into the appliance by storing it in the data store306 of unstructured data. The acts of metadata generation and documentingestion preferably operate virtually concurrently in near-real time.It should be noted that the document metadata 1650 can optionally bestored in a structured database external to the appliance 200.

FIG. 17 depicts this document ingest pre-processing 1700 as a logicalflow. At step 1, the user interacts with GUI 1600 to identify a newdocument 600 for ingestion into appliance 200. GUI 1600 may optionallybe configured to allow the user to specify what metadata is to begenerated from document 600. Next, at step 2, the document 600 isretrieved from its original source location (either an enterprisedocument store 308, the Internet or some other network that isaccessible to the enterprise network 420). Then, firmware 350 performsits document metadata generation operation 604 on document 600 toproduce document metadata 1650. At step 3, the document 600 is thenstored in the file system of data store 306, and the document metadata(including its location in the file system of data store 306) is savedinto relational tables of the RDBMS 304. FIG. 18 illustrates this dataflow superimposed over the appliance 200 of FIG. 2( c).

In this manner, appliance 200 can now use the document metadata 1650indexed by RDBMS 304 to aid decisions as to which documents should besubjected to a query-specified data processing operation (e.g., afull-text searching operation) via coprocessor 450. Furthermore, becausestandardized RDBMS technology has been leveraged within appliance 200,standardized database queries that are well-known to a number of userscan be used to decide which documents are to be subjected to thecoprocessor-based data processing operation when processing a givenquery. This type of query processing is disclosed in theabove-referenced and incorporated U.S. patent application Ser. No.______, filed this same day, entitled “Method and System for HighPerformance Integration, Processing and Searching of Structured andUnstructured Data Using Coprocessors” (identified as Thompson CoburnAttorney Docket Number 44826-72530).

While in the preferred embodiment disclosed herein the coprocessor 450comprises a reconfigurable logic device 402 such as an FPGA, it shouldbe noted that the coprocessor 450 can be realized using other processingdevices. For example, the coprocessor 450 may comprise graphicsprocessor units (GPUs), general purpose graphics processors, chipmulti-processors (CMPs), dedicated memory devices, complex programmablelogic devices, application specific integrated circuits (ASICs), andother I/O processing components. Moreover, it should be noted thatappliance 200 may employ a plurality of coprocessors 450 in either orboth of a sequential and a parallel multi-coprocessor architecture.

While the present invention has been described above in relation to itspreferred embodiments, various modifications may be made thereto thatstill fall within the invention's scope. Such modifications to theinvention will be recognizable upon review of the teachings herein.Accordingly, the full scope of the present invention is to be definedsolely by the appended claims and their legal equivalents.

1. An indexing method comprising: streaming data through a coprocessor;generating metadata for the streaming data using the coprocessor; andgenerating an index for the streaming data based on the generatedmetadata.
 2. The method of claim 1 wherein the coprocessor comprises areconfigurable logic device.
 3. The method of claim 2 wherein thereconfigurable logic device has firmware deployed thereon, wherein thefirmware is configured to perform the metadata generating step.
 4. Themethod of claim 3 wherein the firmware is further configured to performthe index generating step.
 5. The method of claim 4 wherein the indexgenerating step comprises generating a plurality of indexes based on thegenerated metadata.
 6. The method of claim 5 wherein the metadatagenerating step comprises performing an exact matching operation on thestreaming data that is keyed by a plurality of words in a plurality ofdictionaries, wherein each dictionary corresponds to a different one ofthe generated indexes.
 7. The method of claim 6 wherein the metadatagenerating step further comprises performing a regular expressionpattern matching operation on the streaming data that is keyed by aplurality of pre-defined patterns, wherein each pre-defined patterncorresponds to a different one of the generated indexes such that aplurality of the generated indexes correspond to a plurality of thedictionaries and another plurality of the generated indexes correspondto a plurality of the pre-defined patterns.
 8. The method of claim 5wherein the metadata generating step comprises performing an approximatematching operation on the streaming data that is keyed by a plurality ofwords in a plurality of dictionaries, wherein each dictionarycorresponds to a different one of the generated indexes.
 9. The methodof claim 8 wherein the metadata generating step further comprisesperforming a regular expression pattern matching operation on thestreaming data that is keyed by a plurality of pre-defined patterns,wherein each pre-defined pattern corresponds to a different one of thegenerated indexes such that a plurality of the generated indexescorrespond to a plurality of the dictionaries and another plurality ofthe generated indexes correspond to a plurality of the pre-definedpatterns.
 10. The method of claim 5 wherein the metadata generating stepfurther comprises performing a regular expression pattern matchingoperation on the streaming data that is keyed by a plurality ofpre-defined patterns, wherein each pre-defined pattern corresponds to adifferent one of the generated indexes.
 11. The method of claim 5wherein the metadata generating step comprises performing aclassification operation on the streaming data.
 12. The method of claim5 wherein the metadata generating step comprises performing a histogramgeneration operation on the streaming data.
 13. The method of claim 12further comprising performing a clustering operation on the streamingdata based on the generated histogram.
 14. The method of claim 5 furthercomprising performing feature vector extraction on the streaming datausing the firmware.
 15. The method of claim 14 wherein the featurevector extraction operation comprises performing a word parsingoperation on the streaming data.
 16. The method of claim 15 wherein thefeature vector extraction operation further comprises performing a stoplist filter operation on the streaming parsed data.
 17. The method ofclaim 16 wherein the feature vector extraction operation furthercomprises performing a stemming operation on the streaming parsedfiltered data.
 18. The method of claim 5 wherein the streaming datacomprises unstructured data.
 19. The method of claim 18 wherein theunstructured data comprises a plurality of documents.
 20. The method ofclaim 5 further comprising merging the generated indexes with aplurality of operational indexes stored by a relational database.
 21. Amethod of generating metadata for data, the method comprising: receivingdata; streaming the received data through a processing device other thana main processor for the system; and performing a metadata generationoperation on the streaming data using the processing device to therebygenerate metadata for the streaming data.
 22. The method of claim 21wherein the processing device comprises a coprocessor.
 23. The method ofclaim 22 wherein the coprocessor comprises a reconfigurable logicdevice.
 24. The method of claim 23 wherein the reconfigurable logicdevice has firmware deployed thereon, the firmware being configured toperform the metadata generation operation.
 25. The method of claim 22wherein the step of performing the metadata generation operationcomprises generating an index for the streaming data based on thegenerated metadata.
 26. The method of claim 25 wherein the indexgenerating step comprises concurrently generating a plurality of indexesfor the streaming data based on the generated metadata.
 27. The methodof claim 25 wherein the data stream comprises a plurality of data words,and wherein the step of performing the metadata generation operationcomprises: performing an exact match operation between the streamingdata words and a plurality of words in a dictionary; and in response tofinding an exact match between a streaming word and a word in thedictionary, updating an entry in the generated index corresponding tothat dictionary word with a pointer to the data word within thestreaming data that is an exact match to that dictionary word.
 28. Themethod of claim 27 wherein the step of performing the metadatageneration operation comprises concurrently generating a plurality ofindexes for the streaming data based on the generated metadata, whereinthe exact match operation performing step comprises performing the exactmatch operation between the streaming data words a plurality of words ina plurality of dictionaries, and wherein each of the dictionariescorresponds to a different one of the generated indexes.
 29. The methodof claim 28 wherein the index generating step further comprisesgenerating a general index for words within the streaming data that arenot exact matches to any of the words within any of the dictionaries.30. The method of claim 28 wherein at least one of the dictionariescomprises a full English language dictionary.
 31. The method of claim 28wherein at least one of the dictionaries comprises a full foreignlanguage dictionary.
 32. The method of claim 28 wherein at least one ofthe dictionaries comprises a technical dictionary.
 33. The method ofclaim 28 wherein at least one of the dictionaries comprises a medicaldictionary.
 34. The method of claim 27 wherein the dictionary comprisesa list of pre-defined words.
 35. The method of claim 34 furthercomprising changing the content of the dictionary.
 36. The method ofclaim 25 wherein the data stream comprises a plurality of data words,and wherein the step of performing the metadata generation operationcomprises: performing an approximate match operation between thestreaming data words and a plurality of words in a dictionary; and inresponse to finding an exact match between a streaming word and a wordin the dictionary, updating an entry in the generated indexcorresponding to that dictionary word with a pointer to the data wordwithin the streaming data that is an approximate match to thatdictionary word.
 37. The method of claim 36 wherein the step ofperforming the metadata generation operation comprises concurrentlygenerating a plurality of indexes for the streaming data based on thegenerated metadata, wherein the approximate match operation performingstep comprises performing the approximate match operation between thestreaming data words and a plurality of words in a plurality ofdictionaries, and wherein each of the dictionaries corresponds to adifferent one of the generated indexes.
 38. The method of claim 37wherein the index generating step further comprises generating a generalindex for words within the streaming data that are not approximatematches to any of the words within any of the dictionaries.
 39. Themethod of claim 37 wherein at least one of the dictionaries comprises afull English language dictionary.
 40. The method of claim 37 wherein atleast one of the dictionaries comprises a full foreign languagedictionary.
 41. The method of claim 37 wherein at least one of thedictionaries comprises a technical dictionary.
 42. The method of claim37 wherein at least one of the dictionaries comprises a medicaldictionary.
 43. The method of claim 36 wherein the dictionary comprisesa list of pre-defined words.
 44. The method of claim 43 furthercomprising changing the content of the dictionary.
 45. The method ofclaim 25 wherein the data stream comprises a plurality of data words,and wherein the step of performing the metadata generation operationcomprises: performing a regular expression pattern matching operationbetween the streaming data words and a plurality of pre-definedpatterns; concurrently generating a plurality of indexes for thestreaming data based on the generated metadata, wherein each of thepre-defined patterns corresponds to a different one of the generatedindexes; and in response to finding an pattern match between a streamingword and a predefined pattern, updating an entry in the generated indexcorresponding to that predefined pattern with a pointer to the datawithin the streaming data that is a pattern match to that pre-definedpattern.
 46. The method of claim 45 wherein the index generating stepfurther comprises generating a general index for data within thestreaming data that is not a pattern match to any of the pre-definedpatterns.
 47. The method of claim 45 wherein at least one of thepre-defined patterns comprises a credit card number.
 48. The method ofclaim 45 wherein at least one of the pre-defined patterns comprises asocial security number.
 49. The method of claim 45 wherein at least oneof the pre-defined patterns comprises an email address.
 50. The methodof claim 45 wherein at least one of the pre-defined patterns comprises atelephone number.
 51. The method of claim 45 wherein at least one of thepre-defined patterns comprises an Internet uniform resource locator(URL).
 52. The method of claim 45 further comprising changing at leastone of pre-defined patterns.
 53. The method of claim 22 wherein the stepof performing the metadata generation operation comprises performing aclassification operation on the streaming data.
 54. The method of claim53 wherein the classification operation performing step comprisesdetermining whether portions of the streaming data belong to any of aplurality of predefined classes.
 55. The method of claim 54 furthercomprising generating a plurality of data class indexes for thestreaming data based on the classification operation.
 56. The method ofclaim 54 wherein at least one of the pre-defined classes comprises alegal classification.
 57. The method of claim 54 wherein at least one ofthe pre-defined classes comprises a medical classification.
 58. Themethod of claim 54 wherein at least one of the pre-defined classescomprises a news classification.
 59. The method of claim 54 wherein atleast one of the pre-defined classes comprises a sports classification.60. The method of claim 54 wherein at least one of the pre-definedclasses comprises a classification as to whether a data stream portioncontains a pre-defined pattern.
 61. The method of claim 54 wherein thepre-defined pattern comprises a pattern for a credit card number. 62.The method of claim 22 wherein the step of performing the metadatageneration operation comprises generating a histogram for the streamingdata.
 63. The method of claim 62 further comprising performing aclustering operation for the data within the streaming data based on thegenerated histogram.
 64. The method of claim 63 further comprisinggenerating an index for the data within the streaming data based on theclustering operation.
 65. The method of claim 25 further comprising:merging the generated index with an existing operational indexmaintained by a relational database.
 66. The method of claim 25 whereinthe streaming data comprises unstructured data.
 67. The method of claim66 wherein the unstructured data comprises a plurality of documents. 68.The method of claim 25 wherein the streaming data comprises structureddata.
 69. The method of claim 25 further comprising pre-processing thestreaming data with the coprocessor to generate a plurality of featurevectors for the streaming data and using the generated feature vectorswhen generating the index.
 70. A system for generating metadata fordata, the system comprising: a main processor; a processing device otherthan the main processor; wherein the main processor is configured todirect a stream of data to the processing device; and wherein theprocessing device is configured to receive the data stream and perform ametadata generation operation on the data stream to thereby generatemetadata for the data stream.
 71. The system of claim 70 wherein theprocessing device comprises a coprocessor.
 72. The system of claim 71wherein the coprocessor comprises a reconfigurable logic device.
 73. Thesystem of claim 72 wherein the reconfigurable logic device has firmwaredeployed thereon, the firmware being configured to perform the metadatageneration operation.
 74. The system of claim 71 wherein the coprocessoris further configured to generate an index for the data stream based onthe generated metadata.
 75. The system of claim 74 wherein thecoprocessor is further configured to concurrently generate a pluralityof indexes for the data stream based on the generated metadata.
 76. Thesystem of claim 74 wherein the data stream comprises a plurality of datawords, and wherein the coprocessor is configured with an exact matchingmodule, the exact matching module having a dictionary loaded thereon,and wherein the exact matching module is configured to detect any exactmatches that exist between the words in the data stream and a pluralityof words in the dictionary, and wherein the coprocessor is furtherconfigured to, in response to the detection of an exact match between aword in the data stream and a word in the dictionary, update an entry inthe generated index corresponding to that dictionary word with a pointerto the data word within the data stream that is an exact match to thatdictionary word.
 77. The system of claim 76 wherein the coprocessor isfurther configured to concurrently generate a plurality of indexes forthe data stream based on the generated metadata, wherein the exactmatching module is further configured to detect any exact matches thatexist between the words of the data stream and a plurality of words in aplurality of dictionaries, and wherein each of the dictionariescorresponds to a different one of the generated indexes.
 78. The systemof claim 77 wherein the coprocessor is further configured to generate ageneral index for words within the data stream that are not exactmatches to any of the words within any of the dictionaries.
 79. Thesystem of claim 74 wherein the data stream comprises a plurality of datawords, and wherein the coprocessor is configured with an approximatematching module, the approximate matching module having a dictionaryloaded thereon, and wherein the approximate matching module isconfigured to detect any approximate matches that exist between thewords in the data stream and a plurality of words in the dictionary, andwherein the coprocessor is further configured to, in response to thedetection of an approximate match between a word in the data stream anda word in the dictionary, update an entry in the generated indexcorresponding to that dictionary word with a pointer to the data wordwithin the data stream that is an approximate match to that dictionaryword.
 80. The system of claim 79 wherein the coprocessor is furtherconfigured to concurrently generate a plurality of indexes for the datastream based on the generated metadata, wherein the approximate matchingmodule is further configured to detect any approximate matches thatexist between the words of the data stream and a plurality of words in aplurality of dictionaries, and wherein each of the dictionariescorresponds to a different one of the generated indexes.
 81. The systemof claim 80 wherein the coprocessor is further configured to generate ageneral index for words within the data stream that are not approximatematches to any of the words within any of the dictionaries.
 82. Thesystem of claim 74 wherein the data stream comprises a plurality of datawords, wherein the coprocessor is further configured to concurrentlygenerate a plurality of indexes for the data stream based on thegenerated metadata, wherein the coprocessor is configured with a regularexpression pattern matching module, the regular expression patternmatching module having a plurality of pre-defined patterns loadedthereon, and wherein the regular expression pattern matching module isconfigured to detect any pattern matches that exist between the words inthe data stream and the pre-defined patterns, and wherein thecoprocessor is further configured to, in response to the detection of apattern match between data in the data stream and a pre-defined pattern,update an entry in the generated index corresponding to that pre-definedpattern with a pointer to the data within the data stream that is apattern match to that pre-defined pattern.
 83. The system of claim 82wherein the coprocessor is further configured to generate a generalindex for data within the data stream that is not a pattern match to anyof the pre-defined patterns.
 84. The system of claim 71 wherein thecoprocessor is further configured to perform a classification operationon the data stream based on the generated metadata.
 85. The system ofclaim 84 wherein the coprocessor is further configured to determinewhether portions of the data stream belong to any of a plurality ofpredefined classes.
 86. The system of claim 85 wherein the coprocessoris further configured to generate a plurality of data class indexes forthe data stream based on the classification operation.
 87. The system ofclaim 71 wherein the coprocessor is further configured to generate ahistogram for the data stream.
 88. The system of claim 87 wherein theprocessor is further configured to perform a clustering operation forthe data within the data stream based on the generated histogram. 89.The system of claim 74 wherein the processor is further configured tomerge the generated index with an existing operational index maintainedby a relational database.
 90. A method of indexing a plurality of webpages, the method comprising: streaming data for a plurality of webpages through a coprocessor; generating metadata for the streaming datausing the coprocessor; and generating an index for the web pages basedon the generated metadata.
 91. The method of claim 90 wherein thecoprocessor comprises a reconfigurable logic device.
 92. The method ofclaim 91 wherein the reconfigurable logic device has firmware deployedthereon, wherein the firmware is configured to perform the metadatagenerating step.
 93. The method of claim 92 wherein the firmware isfurther configured to perform the index generating step.
 94. The methodof claim 93 wherein the index generating step comprises generating aplurality of indexes based on the generated metadata.
 95. The method ofclaim 94 wherein the metadata generating step comprises performing anexact matching operation on the streaming data that is keyed by aplurality of words in a plurality of dictionaries, wherein eachdictionary corresponds to a different one of the generated indexes. 96.The method of claim 95 wherein the metadata generating step furthercomprises performing a regular expression pattern matching operation onthe streaming data that is keyed by a plurality of pre-defined patterns,wherein each pre-defined pattern corresponds to a different one of thegenerated indexes such that a plurality of the generated indexescorrespond to a plurality of the dictionaries and another plurality ofthe generated indexes correspond to a plurality of the pre-definedpatterns.
 97. The method of claim 94 wherein the metadata generatingstep comprises performing an approximate matching operation on thestreaming data that is keyed by a plurality of words in a plurality ofdictionaries, wherein each dictionary corresponds to a different one ofthe generated indexes.
 98. The method of claim 97 wherein the metadatagenerating step further comprises performing a regular expressionpattern matching operation on the streaming data that is keyed by aplurality of pre-defined patterns, wherein each pre-defined patterncorresponds to a different one of the generated indexes such that aplurality of the generated indexes correspond to a plurality of thedictionaries and another plurality of the generated indexes correspondto a plurality of the pre-defined patterns.
 99. The method of claim 94wherein the metadata generating step further comprises performing aregular expression pattern matching operation on the streaming data thatis keyed by a plurality of pre-defined patterns, wherein eachpre-defined pattern corresponds to a different one of the generatedindexes.
 100. The method of claim 94 wherein the metadata generatingstep comprises performing a classification operation on the streamingdata.
 101. The method of claim 94 wherein the metadata generating stepcomprises performing a histogram generation operation on the streamingdata.
 102. The method of claim 101 further comprising performing aclustering operation on the streaming data based on the generatedhistogram.
 103. The method of claim 94 further comprising performingfeature vector extraction on the streaming data using the firmware. 104.The method of claim 103 wherein the feature vector extraction operationcomprises performing a word parsing operation on the streaming data.105. The method of claim 104 wherein the feature vector extractionoperation further comprises performing a stop list filter operation onthe streaming parsed data.
 106. The method of claim 105 wherein thefeature vector extraction operation further comprises performing astemming operation on the streaming parsed filtered data.
 107. Themethod of claim 94 further comprising merging the generated indexes witha plurality of operational indexes stored by a relational database. 108.A method on enriching a bit stream with metadata, the method comprising:receiving a bit stream corresponding to a data object; generatingmetadata for the bit stream using a coprocessor; and modifying the bitstream based on the generated metadata using the coprocessor.
 109. Themethod of claim 108 further comprising: classifying the data objectbased on the generated metadata using the coprocessor; and wherein themodifying step comprises modifying the bit stream based on theclassifying step using the coprocessor.
 110. The method of claim 109wherein the classifying step comprises assigning a classification to thedata object, wherein the modifying step comprises adding at least onebit to the bit stream that is indicative of the data object's assignedclassification.
 111. The method of claim 110 wherein the coprocessorcomprises a reconfigurable logic device.
 112. The method of claim 111wherein the metadata generating step comprises performing an exactmatching operation on the bit stream that is keyed by a plurality ofwords in a plurality of dictionaries.
 113. The method of claim 111wherein the metadata generating step comprises performing a regularexpression pattern matching operation on the bit stream that is keyed bya plurality of pre-defined patterns.
 114. The method of claim 111wherein the metadata generating step comprises performing an approximatematching operation on the bit stream that is keyed by a plurality ofwords in a plurality of dictionaries.
 115. Within a system comprising amain processor and a coprocessor, a method for building a metadata indexfor unstructured data, the method comprising: streaming unstructureddata through the coprocessor; and performing a metadata generationoperation on the streaming unstructured data using the coprocessor tothereby generate metadata about the unstructured data.
 116. The methodof claim 115 further comprising: storing the unstructured data in a datastore of unstructured data after it has streamed through thecoprocessor; and storing the generated metadata in a database ofstructured data.
 117. The method of claim 116 wherein the coprocessorcomprises a reconfigurable logic device.
 118. The method of claim 117wherein the reconfigurable logic device has firmware deployed thereon,the firmware configured to perform the metadata generation operation.119. The method of claim 118 wherein the metadata includes anidentification of where the unstructured data is stored in the datastore of unstructured data.
 120. The method of claim 119 wherein themetadata storing step comprises storing the metadata in a RDBMS. 121.The method of claim 119 wherein the streaming step comprises: streamingall of an enterprise's unstructured data through the firmware to therebygenerate metadata about that unstructured data.
 122. A system forbuilding a metadata index for unstructured data, the system comprising:a main processor; and a coprocessor configured to receive a stream ofunstructured data and perform a metadata generation operation on theunstructured data stream to thereby generate metadata about theunstructured data.
 123. The system of claim 122 wherein the coprocessorcomprises a reconfigurable logic device.
 124. The system of claim 123wherein the reconfigurable logic device has firmware deployed thereon,the firmware configured to perform the metadata generation operation.125. A method for building a metadata index for unstructured data, themethod comprising: streaming unstructured data through firmware deployedon a reconfigurable logic device, wherein the firmware is configured toperform a metadata generation operation on the unstructured datastreamed therethrough to thereby generate metadata about theunstructured data; storing the unstructured data in a data store ofunstructured data; and storing the metadata about the unstructured datain a database of structured data.
 126. An indexing apparatus comprising:a reconfigurable logic device configured to receive a stream of data,wherein the reconfigurable logic device comprises a firmware pipeline,wherein the firmware pipeline is configured to index the data stream athardware speeds.
 127. The apparatus of claim 126 wherein the firmwarepipeline comprises an exact matching module that operates on the datastream, wherein the exact matching module is keyed with a plurality ofdictionaries to thereby generate indexing information for a plurality ofindexes.
 128. The apparatus of claim 126 wherein the firmware pipelinecomprises an approximate matching module that operates on the datastream, wherein the approximate matching module is keyed with aplurality of dictionaries to thereby generate indexing information for aplurality of indexes.
 129. The apparatus of claim 126 wherein thefirmware pipeline comprises an regular expression pattern matchingmodule that operates on the data stream, wherein the regular expressionpattern module is keyed with a plurality of pre-defined patterns tothereby generate indexing information for a plurality of indexes. 130.The apparatus of claim 129 wherein the firmware pipeline furthercomprises a matching module that operates on the data stream, whereinthe matching module is keyed with a plurality of dictionaries to therebygenerate indexing information for another plurality of indexes.